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Abstract 


To  a  large  extent,  a  system’s  software  architecture  determines  the  quality  attributes  of  both 
the  software  and  the  entire  system.  It  is  also  one  of  the  earliest  artifacts  available  for 
evaluation.  For  a  Department  of  Defense  (DoD)  or  government  acquisition  organization,  the 
ability  to  evaluate  software  architectures  early  in  the  acquisition  cycle  can  positively  affect 
the  delivered  system.  To  assist  a  government  organization  in  evaluating  architectures,  a  series 
of  Quality  Attribute  Workshops  (QAWs)  were  planned  and  an  initial  set  conducted  as  part  of 
a  competitive  acquisition  of  a  complex,  integrated  command  and  control  system.  The  QAW 
is  a  “lightweight”  (i.e.,  non-intrusive)  version  of  the  Architecture  Tradeoff  Analysis 
Method^"^  (ATAM^^)  developed  by  the  Software  Engineering  Institute  (SEI). 

The  QAWs  provided  the  acquiring  government  agency  with  a  means  to  evaluate  each 
contractor’s  software  architectural  approach  and  determine  whether  it  satisfied  the  system’s 
quality  attribute  requirements  (e.g.,  performance,  interoperability,  security).  Since  the 
acquisition  is  ongoing,  follow-on  workshops  are  currently  being  scheduled  to  evaluate  the 
architectural  designs  as  they  evolve. 

This  technical  note  provides  an  overview  of  the  QAW  process  and  the  results  of  the  first  set 
of  workshops,  including  the  perceived  benefits  of  the  workshops  to  both  the  acquirer  and  the 
contractors.  It  also  discusses  future  opportunities  for  applying  a  full-scale  architecture 
evaluation  (e.g.,  an  ATAM  evaluation)  in  early  stages  of  system  acquisition,  and  identifies 
the  benefits  that  could  be  obtained. 


Architecture  Tradeoff  Analysis  Method  and  ATAM  are  service  marks  of  Carnegie  Mellon 
University. 


CMU/SEI-2000-TN-010 


IX 


1  Introduction 


Modem  defense  and  tactical  systems  rely  heavily  on  software  to  deliver  functionality  and 
operational  capabilities. 

The  software  architecture  of  these  systems  is  key  to  achieving— or  failing  to  achieve— their 
quality  attribute  goals.  The  ability  to  evaluate  software  architectures  can  help  ensure  that  the 
delivered  systems  will  meet  these  goals. 

This  technical  note  describes  a  series  of  Quality  Attribute  Workshops  (QAWs)  that  are  being 
conducted  on  behalf  of  a  government  agency  during  its  competitive  acquisition  of  a  complex, 
tactical,  integrated  command  and  control  system.  The  workshops  are  enabling  the  acquiring 
government  agency  to  better  understand  each  contractor’s  proposed  software  design  approach. 
The  workshops  are  also  allowing  the  agency  to  evaluate  the  contractors’  architecture 
development  efforts  very  early  in  the  acquisition  cycle. 

This  technical  note  provides  background  information  on  the  acquisition  program,  including 
the  type  of  system  being  acquired  and  the  acquisition  context  for  conducting  the  workshops. 
Next,  it  describes  the  importance  of  architecture  evaluation  in  system  acquisition  and  its 
relationship  to  a  QAW.  It  then  conveys  the  motivation  for  a  QAW,  describes  how  the 
workshops  are  being  conducted,  and  shows  the  perceived  benefits  to  the  acquiring  agency  and 
the  participating  contractors.  Finally,  the  technical  note  describes  how  architecture 
evaluations  could  be  applied  in  later  phases  of  the  acquisition  process. 
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2  System  Acquisition  Context 


2.1  The  Acquisition  Organization  and  the  System  Being  Acquired 

This  case  study  describes  an  ongoing  acquisition.  The  identities  of  the  participants  have  been 
disguised  to  protect  the  privacy  of  all  those  involved.  The  acquiring  government  agency  is 
referred  to  as  the  “AGA”  and  the  system  being  acquired  as  the  “TIC”  system — an  elaborate 
and  sophisticated,  tactical  integrated  command,  control,  and  communication  system. 

Figure  1  shows  the  TIC  system  concept.  It  is  a  true  “system  of  systems.”  It  includes  multiple 
ground,  air,  sea,  and  space  assets  for  conducting  a  prescribed  set  of  missions  in  different 
localities. 


Figure  1:  Conceptual  Overview  of  the  TIC  System 

The  TIC  system  must  concurrently  support  missions  involving  different  asset  combinations  in 
predictable  (and  unpredictable)  tactical  situations  and  environmental  conditions.  As  a  result, 
the  contractual  system  specification  includes  quality  attribute  requirements  (e.g.,  security, 
interoperability,  performance)  that  reflect  this  advanced  system’s  capabilities.  Of  course,  the 
logistical  requirements  must  also  be  considered.  The  AGA  faced  the  challenge  of  evaluating  a 
contractor’s  proposed  design  to  see  if  it  provided  the  required  quality  attributes. 
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2.2  The  Acquisition  Strategy 

Since  a  huge  investment  in  time  and  resources  was  involved,  the  AGA  adopted  a  two-phase 
acquisition  strategy  (shown  in  Figure  2)  with  a  “rolling  down-select”'  to  maximize 
competition  and  reduce  risk. 


Phase  1 
Contract 
Awards 


Phase  1 


RFP 

Issued  for 
Phase  2 


Two  Study  Contracts 
Competitively  Awarded 


Develop 

Conceptual 

Design 


Evaluatio 
of 

_ Pontract 

Develop  D^liverabl^ 
Architectural  i 
Design  * 


Phase  2 


Down-Select  for  Single 
System  Development  Contract 


Technical  j 

Phase  2  j 

Proposal  I 

Source  j 

Evaluation  j 

Selection  i 

^etitive  Award 
of 

System  Development 
and  Inplementation 
Contract 


Figure  2:  Two-Phase  System  Acquisition  Strategy 

In  Phase  1,  the  agency  competitively  awarded  two  contractors  fixed-price  study  contracts  to 
define  their  systems  at  a  very  high  level.  The  Phase  1  study  contract,  which  is  currently 
ongoing,  calls  for  each  contractor  to  develop  a  conceptual  design  for  the  system  followed  by 
an  architectural  design.  In  parallel  with  the  architectural  design,  the  contractor  must  also 
develop  a  Concept  of  Operations  (CONOPS)  for  the  TIC  system.  The  study  contract  enables 
both  contractors  to  refine  their  system  concepts  and  architectures  so  that  they  can  estimate 
system  development  costs  and  schedules  with  known  (and  acceptable)  risk.  At  the  time  the 
first  set  of  workshops  was  conducted,  both  contractors  had  completed  their  conceptual 
designs  and  were  developing  their  architectural  designs. 


The  contractors  were  not  scheduled  to  complete  their  architectural  designs  and  formally 
deliver  them  to  the  AGA  until  the  end  of  the  second  half  of  Phase  1.  The  study  contracts 
specify  that  the  contractors  must  develop  their  architectures  in  accordance  with  the 
Department  of  Defense’s  Command,  Control,  Communication,  Computers,  Intelligence, 
Surveillance,  and  Reconnaissance  (C4ISR)  Architecture  Framework^  [AWG  98].  As  a  result, 
each  contractor  must  deliver  C4ISR  operational,  systems,  and  technical  architecture 


'  A  “rolling  down-select”  refers  to  an  acquisition  strategy  where  relatively  short-term  contracts  are 
initially  awarded  to  multiple  contractors  followed  by  another  Request  for  Proposal  (RFP)  to 
competitively  award  a  single  contract  (to  complete  the  work)  to  the  contractor  submitting  the  “best 
value”  proposal. 

^  This  framework  is  becoming  the  required  method  for  describing  information  systems  architectures 
within  the  DoD  and  other  U.S.  government  agencies. 
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descriptions^  for  their  proposed  systems.  Seven  C4ISR  operational  views  define  operations 
from  asset  and  user  perspectives.  Eleven  C4ISR  systems  products  define  the  organization  of 
hardware  and  software  components.  Two  C4ISR  technical  products  identify  the  standards  and 
commercial  products  in  the  system,  forecast  their  evolution,  and  describe  the  inclusion  of 
emerging  standards  and  commercial  products. 

Integrated  Project  Teams  (IPTs)  are  currently  evaluating  the  Phase  1  deliverables  to  determine 
whether  they  meet  the  contractual  requirements.  The  IPTs  are  also  assessing  the  strengths, 
weaknesses,  and  risks  of  each  contractor’s  proposed  approach.  A  separate  Architecture  EPT  is 
evaluating  each  contractor’s  proposed  architecture.  Among  other  requirements,  the  Phase  I 
contract  specifically  states  that  the  TIC  system  must  satisfy  five  system  quality  attributes: 
performance,  availability,  security,  interoperability,  and  modifiability. 

Once  the  Phase  I  study  contracts  are  complete,  the  AGA  will  begin  Phase  2  and  issue  a 
Request  for  Proposal  (RFP).  The  AGA  team  will  then  formally  evaluate  the  contractors’  Phase 
2  proposals.  A  source-selection  team  will  make  a  “down-select”  and  award  the  TIC  system 
development  and  implementation  contract  to  the  organization  whose  proposal  represents  the 
“best  value”  to  the  government. 
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An  SEI  Technical  Report  (CMU/SE1-99-TR-014)  “Architecture  Tradeoff  Analyses  ofCdlSR 
Products”  describes  how  C4ISR  products  can  be  used  for  generating  quality  attribute-specific 
scenarios  in  the  context  of  an  ATAM  evaluation.  [Barbacci  99] 
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3  Role  of  Architecture  Evaluation  in  System  Acquisition 


Software  architecture  is  important  because  it  embodies  the  decisions  and  tradeoffs  made 
during  the  earliest,  high-level  design  stages.  These  design  decisions  will  drive  the  entire 
software  development  effort  and  ultimately  determine  software  quality.  These  decisions  are 
the  hardest  to  get  right.  They  have  the  farthest-reaching  repercussions  on  the  system’s 
operation,  capabilities,  and  qualities.  These  decisions  are  also  the  hardest  to  change 
downstream.  If  an  inappropriate  architectural  choice  is  made,  the  impact  is  profound.  Studies 
show  that  fixing  an  error  during  requirements  or  early  design  phases  costs  orders  of 
magnitude  less  than  fixing  the  same  error  found  during  testing  [Boehm  81].  Thus,  it  makes 
sense  to  evaluate  the  software  architecture  of  a  system  as  early  as  possible. 

For  example,  if  a  system  has  stringent  real-time  performance  requirements,  the  architect  must 
pay  attention  to  inter-component  communication  and  intra-component  deadlines.  If  there  are 
modifiability  goals,  the  architect  must  pay  attention  to  the  encapsulation  properties  of 
components.  If  reliability  is  important,  the  architect  must  pay  attention  to  redundant 
components.  The  list  goes  on.  In  each  case,  the  quality  attribute  can  be  traced  back  to  the 
decomposition  of  the  total  system  into  parts  and  the  ways  in  which  those  parts  communicate 
and  cooperate  with  each  other.  While  a  “good”  architecture  cannot  guarantee  a  successful 
implementation  (i.e.,  the  system  meets  its  quality  goals),  a  “bad”  architecture  can  certainly 
preclude  one. 

Ideally,  risk  mitigation  should  begin  during  architecture  definition  and  refinement.  An 
architecture  evaluation''  is  one  risk  mitigation  activity  that  has  been  shown  to  have  a  high 
payoff.  While  conducting  an  architecture  evaluation  may  appear  to  be  an  obvious  step,  it 
certainly  isn’t  a  routine  occurrence,  especially  in  DoD  and  government  organizations  that 
greatly  depend  on  acquisition  practices. 

The  Architecture  Tradeoff  Analysis  Method  (ATAM)  is  a  technique  for  analyzing  a  software 
architecture  with  respect  to  the  quality  attributes  of  the  system.  The  technical  staff  at  the  SEI 
have  developed  and  refined  this  method  over  the  past  three  years  [Kazman  00].  The  ATAM 
can  detect  areas  of  potential  risk  within  the  architecture  of  a  complex  software-intensive 
system.  It  reveals  how  well  an  architecture  satisfies  goals  and  provides  insight  into  how  these 
quality  goals  interact  with  each  other.  It  also  allows  engineering  tradeoffs  to  be  made  among 
possibly  conflicting  quality  goals. 


This  is  distinguished  from  an  architectural  review  that  is  a  typical  part  of  an  acquisition  milestone 
such  as  a  Critical  Design  Review. 
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The  ATAM  evaluation  can  be  applied  early  in  the  software-development  life  cycle.  It  can  be 
performed  quickly  and  inexpensively.  And  it  does  not  require  detailed  analyses  of  measurable 
quality  attributes,  such  as  mean  time  to  failure  or  latency,  to  succeed. 

Members  of  the  SEI  technical  staff  have  used  ATAM  to  evaluate  the  software  architectures  of 
systems  at  various  phases  in  their  life  cycles: 

•  before  architectural  decisions  have  been  completely  determined 

•  after  architectural  decisions  have  been  determined,  but  before  detailed  design  and  coding 
activities  have  started  or  have  been  completed 

•  after  system  deployment,  when  modernization  is  being  considered 

•  before  system  development,  when  multiple  candidate  architectures  are  being  considered 

A  complete  description  of  the  ATAM  method  is  found  in  ATAM:  Method  for  Architecture 
Evaluation  [Kazman  00].  Currently,  there  are  no  generally  accepted  industry-wide  standards 
for  describing  an  architecture.  Therefore,  ATAM  evaluations  are  often  tailored  to  the  available 
documentation. 
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4  The  Quality  Attribute  Workshop  (QAW) 


In  essence,  a  QAW  is  a  “lightweight”  or  less  intrusive  version  of  an  ATAM  evaluation.  Like 
an  ATAM  evaluation,  it  does  not  aim  at  an  absolute  measure  of  architectural  quality.  Rather, 
the  objective  is  to  identify 

•  scenarios  from  the  point  of  view  of  a  diverse  group  of  stakeholders 

•  quality  attribute  sensitivity  points,  tradeoffs,  and  risks 

•  possible  mitigation  strategies 

In  a  QAW,  the  actual  analysis  burden  falls  on  the  contractors,  with  the  SEI  facilitating  the 
review  of  the  analysis.  Stakeholders  typically  include  architects,  developers,  managers, 
sponsor  representatives,  systems  and  software  engineers,  logistics  personnel,  end  users,  and 
others  who  have  a  vested  interest  in  the  system. 

In  conducting  a  QAW,  the  workshop  facilitators  depend  on  a  variety  of  inputs  including 
stakeholder  points  of  view,  architecture  documentation,  and  architectural  designs.  The 
contractor  is  responsible  for  supplying  this  information.  Prior  to  a  workshop,  the  participants 
receive  a  QAW  handbook,  similar  to  the  Quality  Attribute  Workshop  Participants  Handbook 
[Barbacci  01].  It  describes  QAW  activities  and  the  facilitation  tools  that  will  be  used.  The 
workshops  are  typically  one  and  a  half  days  in  length. 

The  QAW  process  is  used  to  discover  and  document  quality  attribute  risks,  sensitivity  points, 
and  tradeoffs,  where 

•  Quality  attribute  risks  are  architectural  decisions  that  might  create  future  problems  for 
some  quality  attribute  requirement. 

•  Sensitivity  points  are  architectural  parameters  for  which  a  slight  change  makes  a 
significant  difference  in  some  quality  attribute. 

•  Tradeoffs  are  architectural  parameters  affecting  more  than  one  quality  attribute. 
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Figure  3:  Roadmap  of  Activities  for  Quality  Attribute  Workshop 

As  shown  in  Figure  3,  the  QAW  process  consists  of  four  major  activities:  (1)  Scenario 
Generation  and  Prioritization,  (2)  Scenario  Analysis,  (3)  Tradeoff  and  Risk  Identification,  and 
(4)  Decision  Making. 

Scenario  Generation  occurs  during  a  facilitated  brainstorming  process.  Stakeholders  propose 
scenarios  that  test  the  effectiveness  of  the  contractor’s  proposed  C4ISR  architecture'  to 
achieve  specific  quality  attributes  within  a  specific  mission  and  geographic  context.  These 
scenarios  are  candidates  for  use  in  exercising  the  architecture  against  current  and  future 
situations.  In  general,  there  are  three  types  of  scenarios:  (1)  use-case  scenarios,  (2)  growth 
scenarios,  and  (3)  exploratory  scenarios. 

In  the  Prioritization  activity,  stakeholders  are  assigned  a  number  of  votes  that  they  can 
allocate.  The  five  or  six  scenarios  gamering  the  most  votes  are  selected  for  further  analysis. 

During  the  Scenario  Analysis,  stakeholders  choose  an  appropriate  architectural  style  or 
architecture  fragment,  and  apply  the  scenario  to  the  artifact.  This  analysis  is  designed  to 
identify  important  architectural  decisions  and  sensitivity  points.  As  a  result  of  this  activity,  the 
stakeholders  might  decide  to  conduct  additional,  more  detailed  or  formal  analyses  of  the 
scenarios  or  artifacts.  These  activities  take  place  offline,  not  during  the  workshop. 

During  Tradeoff  and  Risk  Identification,  stakeholders  use  the  results  of  the  analysis  to  identify 
and  document  risks  (i.e.,  potential  future  problems  that  might  impact  cost,  schedule,  or  quality 
attributes  of  the  system). 


^  From  the  perspective  of  the  Architecture  IPT,  the  systems  and  technical  levels  of  the  C4ISR 
architecture  are  the  primary  focus  of  the  workshops;  the  operational  level  is  viewed  as  setting  the 
context  and  background  that  bounds  the  scope  of  the  architecture  evaluation. 
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During  this  phase,  fruitful  scenarios  to  consider  include 


•  a  single  scenario  that  involves  two  quality  attributes  explicitly  or  implicitly 

•  multiple  scenarios  about  different  quality  attributes  sharing  common  factors  (e.g., 
resources,  protocols) 

•  multiple  conflicting  scenarios 

In  the  final  activity.  Decision  Making,  contractor  management  adjudicates  the  tradeoffs  and 
risks.  These  decisions  are  typically  guided  by  contract  requirements  and  system  deliverables, 
including  the  prescribed  quality  attributes,  the  proposed  system  concept  of  operations,  and  the 
contractor’s  business  goals  and  interests.  Other  upper-level  managers  may  be  brought  in  and 
advised  of  high-visibility  alternatives  and  the  corresponding  impact  of  changes.  This 
information  is  often  sent  to  the  sponsor  as  well. 

QAWs  enable  an  organization  to  generate  and  analyze  scenarios  about  a  system  that  is  still  in 
the  process  of  being  defined.  This  does  not  obviate  the  need,  however,  for  something  concrete 
to  analyze.  For  example,  if  a  scenario  suggests  that  message  throughput  is  important,  QAW 
team  members  need  a  sketch  of  the  components  and  connections  that  implement  the 
subsystem  that  processes  the  messages.  Since  the  workshop  team  members  don’t  expect  such 
decisions  to  have  been  made  when  they  analyze  a  scenario,  the  architect  can  suggest  the 
reasonable  or  likely  candidate  architecture  for  purposes  of  the  exercise.  The  stakeholders  are 
not  bound  to  that  solution  and  are  not  “graded”  on  the  effectiveness  of  a  choice  made  on  the 
spur  of  the  moment.  The  scenarios,  screening  and  exploratory  questions,  and  attribute  tables 
remain  with  the  organization,  and  the  developers  can  repeat  the  exercise  using  alternative 
subsystem  architectures. 
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5  Motivation  for  Conducting  a  QAW 


In  the  case  under  discussion,  the  AGA  did  not  have  an  effective  means  for  evaluating  whether 
a  contractor’s  proposed  design  provided  the  required  system  quality  attributes.  Additionally, 
there  were  no  contractual  provisions  to  permit  the  AGA  to  conduct  a  formal  architecture 
evaluation.  Instead,  the  two  contractors  were  only  required  to  deliver  documentation 
describing  their  C4ISR  architectures.  Moreover,  conducting  an  ATAM  evaluation  at  this 
juncture  was  considered  inappropriate  because  the  contractors  were  just  developing  their 
architectures  and  were  not  prepared  for  a  formal  evaluation. 

As  a  result,  the  AGA  tasked  an  SEI  team  to  develop  and  conduct  a  series  of  incremental 
QAWs  under  the  purview  of  the  Architecture  IPX.  The  goal  of  the  workshops  was  to  provide  a 
suitable  forum  for  discussing  and  evaluating  quality  attributes. 

Plans  for  the  QAWs  included  conducting  three  workshops  at  each  contractor’s  site.  The 
workshops  were  scheduled  during  the  architectural  design  portion  of  the  Phase  1  study 
contracts,  prior  to  the  Phase  2  “rolling  down-select.’’  Figure  4  shows  the  workshop  schedule 
relative  to  the  overall  system  acquisition  cycle. 
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Figure  4:  Scheduled  Phasing  of  Quality  Attribute  Workshops 
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The  AGA’s  overall  objectives  for  the  first  set  of  QAWs  were  to 

•  gain  a  better  understanding  of  each  contractor’s  conceptual  design  and  proposed 
architecture  and  its  ability  to  provide  the  system  quality  attributes 

•  introduce  contractors  to  the  technology  available  for  evaluating  architectural 
representations 

•  enable  participating  contractors  to  gain  insight  into  AGA  issues  and  priorities 

•  provide  a  common  basis  for  continued  and  closer  dialogue  during  the  study  phase 

5.1  Ground  Rules  Adopted  for  Conducting  the  Workshops 

Although  the  AGA  encouraged  the  contractors  to  participate  in  these  QAW s,  their 
participation  was  strictly  voluntary.  Prior  to  conducting  each  workshop  at  the  contractor’s  site, 
AGA  representatives  made  it  clear  that 

•  Participation  is  not  a  contractual  requirement. 

•  Each  contractor  may  decide  whether  to  continue  and  how. 

•  All  attendees  are  either  contractor  representatives  or  AGA  and  SEI  representatives  who 
have  signed  a  Non-Disclosure  Agreement  (NDA). 

•  The  initial  workshop  approach  will  be  the  same  for  both  contractors,  but  follow-up 
sessions  will  be  tailored  per  their  desires. 

•  The  QAWs  will  focus  on  the  architecture  evaluation  process. 

The  SEI  team  codified  the  technology  and  facilitated  the  workshops  under  the  sponsorship  of 
the  Architecture  IPT. 

Although  the  QAWs  were,  and  still  are,  being  conducted  concurrently  with  the  technical 
assessment  of  Phase  1  deliverables,  the  two  efforts  remain  separate  in  keeping  with  the 
ground  rules  of  the  workshops. 
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6  Workshop  Results  and  QAW  Lessons  Learned 


With  the  voluntary  cooperation  of  the  contractors,  good  things  happened  in  the  first  set  of 
workshops.  The  workshops  enabled  the  AGA  to  establish  a  proactive  means  of  working  with 
the  contractors  to  conduct  architecture  evaluations  in  the  early  stages  of  system  design.  And 
the  results  established  a  solid  “analysis  baseline”  that  the  AGA  and  the  contractors  can  build 
on  in  future  workshops  to  fully  evaluate  the  architectures’  inherent  quality  attribute 
sensitivities,  tradeoffs,  and  risks. 

Although  it  was  not  practical  to  complete  all  the  roadmap  activities,  the  QAW  team  did 
successfully  generate  a  set  of  scenarios,  prioritize  them,  and  conduct  a  cursory  analysis  of  the 
six  top-priority  scenarios.  The  QAW  team  did  not  perform  the  Tradeoff  and  Risk 
Identification  Activity,  however,  due  to  a  lack  of  time  and  system  definition.  Instead,  team 
members  evaluated  the  top-ranked  scenarios  and  identified  likely  attribute  tradeoffs  and 
sensitivity  points.  From  the  standpoint  of  the  AGA,  the  bottom  line  was  that  all  parties  gained 
from  the  workshops. 

From  the  AGA’s  perspective,  the  workshops  enabled  it  to 

•  have  an  informal,  but  structured  information  exchange  that  helped  clarify  the  contractors’ 
approach  to  satisfying  the  system  requirements  and  quality  attributes 

•  have  a  more  substantive  dialogue  about  the  contractors’  proposed  operational  concepts 
and  C4ISR  architectural  issues 

•  understand  the  scenarios  of  concern  to  the  contractors  and  the  issues  and  implications 
associated  with  those  scenarios 

•  identify  and  address  stakeholders’  concerns  and  the  degree  to  which  stakeholders  and 
contractors  shared  a  common  view  of  how  the  system  operates 

•  develop  a  set  of  high-priority  scenarios  to  explore  the  quality  attributes  of  its  proposed 
system 

•  examine  some  of  the  contractors’  decision-making  processes  and  evaluate  their  ability  to 
articulate  their  conceptual  designs  and  C4ISR  architectures 

It  became  apparent  during  the  workshops  that  the  contractors,  in  general,  were  still  wrestling 
with  their  operational  concepts  of  how  the  TIC  system  would  function  and  operate  with  the 
spectrum  of  AGA  ground,  air,  space,  and  sea  assets.  In  one  case,  it  was  obvious  that  the 
workshop  represented  the  first  time  that  all  the  stakeholders  were  “on  the  same  page”  about 
operational  issues.  There  were  also  instances  of  the  operational  concept  being  refined  “on  the 
fly.” 
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From  the  contractors’  perspective,  the  workshops  created  a  greater  awareness  of 

•  misunderstandings  among  the  various  stakeholders,  operational  issues  that  remain  to  be 
resolved,  and  design  decisions  that  must  be  revisited 

•  the  need  to  work  on  communication  among  project  personnel 

•  the  value  of  using  scenarios  to  exercise  the  system,  drive  it  down  to  the  architectural 
level,  and  determine  its  impact  on  the  system’s  quality  attributes 

•  the  need  to  capture  issues  and  concerns  that  were  particularly  important  to  the  AGA  and 
the  architecture  IPT  and  to  identify  where  there  was  a  lack  of  understanding  on  their  part 

As  a  result,  the  AGA  is  planning  follow-on  workshops  to  help  the  Architecture  IPT  gain  a 
better  understanding  of  the  systems  being  proposed.  The  second  set  of  workshops  will  refine 
and  apply  scenarios  generated  by  the  AGA  to  evaluate  the  preliminary  architectures  of  the 
competing  contractors.  The  last  set  of  workshops  will  evaluate  each  contractor’s  final 
architecture  using  scenarios  selected  by  the  AGA. 

The  QAW  team  members  from  the  SEI  expect  that  communication  between  all  parties  will 
improve  in  subsequent  workshops.  The  contractors  were  reluctant  to  disclose  some  details  of 
their  system  designs.  They  appeared  leery  of  the  workshops  and  possibly  needed  some 
workshop  experience  before  revealing  their  proposed  designs.  This  may  have  been  due,  in 
part,  to  the  “high  stakes”  environment  created  by  this  very  competitive  acquisition.  Other 
factors  limiting  communication  may  have  been  that  their  operational  concepts  were  still 
evolving,  and  that  their  architectures  were  in  very  tentative  stages  of  development. 
(Contractually,  they  were  not  scheduled  to  make  a  delivery  for  several  months.) 

In  addition,  the  QAW  team  members  from  the  SEI  learned  that  they  could  not  cover  all  four 
roadmap  activities  in  a  single  workshop.  They  also  learned  that  the  examples  in  the  workshop 
handbook  were  too  detailed. 

6.1  Acquisition  Issues  Reiated  to  Architecture  Evaiuation 

One  pertinent  acquisition  issue  arose  concerning  the  ground  rules.  Since  the  workshops  were 
“advertised”  as  voluntary  and  informal  interchanges,  it  followed  that  information  or  results 
derived  from  the  QAWs  should  not  be  used  in  the  formal  technical  assessment  of  the  Phase  1 
architecture,  or  in  the  Phase  2  technical  evaluation  and  source-selection  process.  Without  the 
authorization  of  the  Contracting  Officer  or  the  Contracting  Officer’s  Technical  Representative 
(COTR),  any  formal  use  of  the  data  could  result  in  a  protest  and  place  the  acquisition  in 
jeopardy.  This  situation  can  be  avoided  by  proactively  specifying  in  the  contract  that 
workshops  must  be  conducted  as  formal  risk-mitigation  checkpoints. 

Another  acquisition  issue  concerned  the  relationship  of  the  Phase  1  architecture  deliverables 
to  the  Phase  2  contract.  Unless  this  relationship  is  clearly  defined,  the  Phase  1  and  Phase  2 
contractual  efforts  may  not  be  seamless.  Since  Phase  2  has  many  of  the  characteristics  of  an 
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independent  acquisition,  a  contractor  could  submit  a  final  proposal  that  is  based  on  a  new  or 
modified  conceptual  design  and  architecture.  In  fact,  this  may  be  positive  since  the  changes 
may  have  been  incorporated  to  reduce  the  cost  and  time  needed,  and  to  mitigate  risks 
discovered  during  Phase  1.  This  gives  rise,  though,  to  several  thorny  questions: 

•  How  will  the  results  of  the  IPT  evaluation  of  the  Phase  1  architecture  be  used  in  the  Phase 
2  source  selection  if  the  contractor’s  Phase  2  technical  proposal  affects  the  architecture 
proposed  in  Phase  1  ? 

•  How  will  Phase  2  proposals  be  evaluated  comparatively  if  one  of  the  contractors 
significantly  changes  its  proposed  Phase  1  architecture  and  another  does  not? 

•  Will  there  be  sufficient  time  and  resources,  and  an  effective  means  to  evaluate  any 
changes  that  affect  the  architecture? 

One  potential  remedy  is  a  requirement  in  the  Phase  2  RFP  that  each  contractor  identify  the 
relationship  of  Phase  1  deliverables  to  its  proposed  technical  approach  for  Phase  2.  If  the 
proposed  approach  for  Phase  2  differs  in  any  way,  the  contractor  should  describe  the  scope, 
nature,  extent,  and  impact  of  any  changes  that  affect  the  architecture.  Another  possible  remedy 
is  to  perform  an  architecture  evaluation  as  part  of  the  Phase  2  source-selection  process  or 
system-development  process. 

The  general  underlying  lesson  learned  is  that  it  is  always  best  to  “plan  early”  to  incorporate 
architecture  evaluations  in  a  system  acquisition.  This  also  applies  to  downstream 
opportunities  to  incorporate  architecture  evaluations  in  Phase  2.  Several  of  those  opportunities 
are  described  in  the  next  section. 
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7  Downstream  Opportunities  for  Conducting  Architecture 
Evaluations 


One  question  worth  considering  is  where  would  formal^  architecture  evaluations  have  the 
greatest  benefit?  Since  an  inappropriate  architectural  choice  can  have  such  a  profound  impact, 
it  makes  sense  to  evaluate  the  architecture  at  critical  points  throughout  the  system’s 
acquisition. 

In  particular,  there  are  two  downstream  opportunities  in  Phase  2  where  conducting  an 
architecture  evaluation  can  potentially  achieve  a  high  payoff.  These  two  points  of  opportunity 
correspond  to  the  technical  proposal  evaluation  prior  to  contract  award  and  early  in  the  system 
development  cycle.  They  are  depicted  in  Figure  5  below. 
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Figure  5:  Key  Acquisition  Points  for  Conducting  Downstream  Architecture 
Evaluations 

The  next  sections  describe  the  two  potential  opportunities  in  detail. 


*  It  is  formal  in  the  sense  that  it  is  a  contractual  requirement. 
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7.1  Formal  Architecture  Walkthrough  and  Evaluation  as  an  Element  of 
Technical  Proposal  Evaluation  and  Source  Selection 

We  recommend  that  each  contractor  prepare  an  architecture  walkthrough  and  evaluation  as 
part  of  its  technical  proposal  presentation  (in  accordance  with  guidelines  supplied  by  the  AGA 
in  the  RFP).  In  the  walkthrough  and  evaluation,  each  competing  contractor  would  present  its 
proposed  architecture  and  show  how  it  satisfies  the  system’s  quality  attribute  requirements  for 
a  prescribed  set  of  scenarios.  This  can  be  an  effective  means  of  evaluating  a  proposed 
software  architecture  and  comparing  one  approach  with  another. 

Bernhardt  describes  using  such  a  walkthrough  and  evaluation  as  part  of  the  source  selection 
process  for  a  major  DoD  system  acquisition  [Bernhardt  00].  In  this  example,  the  architecture 
evaluation  results  were  used  to  select  the  best  value  among  the  proposed  architectures.  This 
involves  including  the  appropriate  language’  in  the  Instructions  to  Offerors  and  in  Section  M 
of  the  RFP  that  describes  the  Evaluation  Factors  for  Award. 

Should  the  AGA  elect  to  require  an  architecture  walkthrough  and  evaluation  in  Phase  2,  it 
should  consider  building  on  the  QAW  results.  As  the  first  step  in  this  strategy,  the  AGA  should 
prepare  and  issue  a  set  of  20  to  40  representative  scenarios*  to  both  the  contractors  far  in 
advance  of  issuing  the  RFP  for  Phase  2.  These  candidate  scenarios  would  include  normal, 
adverse,  and  growth  situations  that  reflect  the  high  priority  mission  needs  and  quality  attribute 
requirements  of  the  AGA. 

Coinciding  with  the  submission  of  the  contractors’  written  technical  proposals,  the  AGA 
would  select  a  small  number  of  scenarios  from  this  larger  group  and  inform  the  contractors 
which  ones  were  selected.  It  would  require  each  contractor  to  conduct  a  formal  architecture 
walkthrough  and  evaluation  using  this  smaller  set  of  scenarios’  as  part  of  its  technical 
proposal  presentation.  This  strategy  forces  a  contractor  to  consider  all  20  to  40  scenarios  in 
order  to  fully  prepare  for  the  required  architecture  walkthrough  and  evaluation  that  will  occur 
during  technical  proposal  evaluation. 


’  This  topic  will  be  covered  in  a  future  technical  note. 

*  The  number  of  scenarios  would  depend,  in  part,  on  the  complexity  of  the  system. 

’  If  the  scenario  evaluation  were  commensurate  with  the  QAW  approach,  a  practical  limit  on  the 
number  of  scenarios  would  be  four. 
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7.2  Conducting  In  Situ  Architecture  Evaluations  During  System  Development 

Once  a  contract  has  been  awarded,  we  recommend  performing  an  in-depth  architecture 
evaluation  (e.g.,  an  ATAM  evaluation).  An  architecture  evaluation  can  help  the  acquiring 
organization 

•  select  an  architecture  among  several  candidate  architectures 

•  evaluate  architecture  designs  to  reduce  program  risks 

•  refine  a  design  once  an  architecture  has  been  chosen 

Bergey  describes  using  in  situ  software  architecture  evaluations  as  contractual  checkpoints  in 
system  acquisitions  [Bergey  99].  These  evaluations  enable  architects  to  address  risks  when 
costs  and  effort  for  later  rework  can  be  minimized.  The  contractor  in  cooperation  with  the 
acquiring  organization  performs  the  software  architecture  evaluations.  This  is  consistent  with 
the  spirit  of  acquisition  reform,  because  the  contractor  is  not  told  how  to  develop  the  system, 
only  what  qualities  it  must  deliver  in  the  system.  It  also  provides  the  government  with  an 
effective  means  of  evaluating  system  quality  and  reducing  risk.  Although  this  technical  note 
refers  to  the  Architecture  Tradeoff  Analysis  Method  for  such  evaluations,  any  evaluation 
method  that  focuses  on  quality  attributes  could  be  used. 

Using  architecture  evaluations  as  contractual  checkpoints  would  enable  the  AGA  to  monitor 
and  evaluate  the  winning  contractor’s  proposed  Phase  2  architecture'”  early  in  the  system- 
development  cycle.  This  could  prevent  major  design  problems  from  rippling  through  the 
entire  software  development  effort.  Again,  these  evaluations  could  also  help  the  AGA  to 
explore  other  potential  risks  and  weaknesses  and  ensure  that  corrective  action  is  taken. 


The  architecture  the  contractor  proposes  for  Phase  2  development  might  be  significantly  different 
than  the  architecture  proposed  during  the  Phase  1  study  contract. 
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8  Summary 


In  this  technical  note,  we  discussed  the  key  role  of  software  architecture  in  enabling  the 
quality  of  software-intensive  systems.  We  also  discussed  how  software  architecture  evaluation 
can  reduce  risk  in  a  system  acquisition  by  clarifying  the  architecture’s  ability  to  meet  quality 
attribute  requirements.  We  described  how  QAWs  are  being  used  in  a  major  system  acquisition 
to  evaluate  architectures  in  a  relatively  non-intrusive  manner,  and  how  they  are  enabling  the 
acquiring  organization  to  generate  and  analyze  scenarios  about  a  system  during  the  design 
process.  We  described  the  major  activities  involved  in  conducting  a  QAW  and  how  these 
activities  can  provide  insight  into  the  contractors’  progress  of  architecture  development.  We 
also  identified  workshop  results,  lessons  learned,  and  acquisition  issues  that  surfaced  as  a 
result  of  developing  and  facilitating  the  workshops.  Finally,  we  identified  two  key 
downstream  opportunities  for  incorporating  a  formal  architecture  evaluation  as  part  of  the 
system  acquisition  strategy. 

Members  of  the  SEI  Product  Line  Systems  Program  are  collaborating  with  several  DoD  and 
government  acquisition  organizations  to  explore  the  appropriate  use  of  QAWs  and  ATAM 
evaluations  within  these  organizations.  The  goal  is  to  help  these  organizations  adopt 
architecture  evaluation  practices  and  to  ensure  that  architecture  evaluation  becomes  an 
integral  part  of  the  acquisition  process. 

To  date,  SEI  staff  members  have  conducted  a  handful  of  QAWs.  As  experience  is  gained,  we 
will  continue  to  share  our  expertise  in  future  technical  notes. 
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Feedback  and  Contact 


Comments  or  suggestions  about  this  document  or  the  series  of  technical  notes  on  architecture 
evaluation  in  the  DoD  are  welcome.  We  want  this  series  to  be  responsive  to  the  needs  of  DoD 
and  government  personnel.  To  that  end,  comments  concerning  this  technical  note,  the 
inclusion  of  other  topics,  or  any  other  issues  or  concerns  will  be  of  great  value  in  continuing 
this  series.  Comments  or  suggestions  should  be  sent  to 

Linda  Northrop,  Director 
Product  Line  Systems  Program 
lmn@sei.cmu.edu 


Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 
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